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Abstract 

Recent experimental studies of living neural networks [5,9] reveal that 
their global activation induced by electrical stimulation can be explained 
using the concept of bootstrap percolation on a directed random network. 
The experiment consists in activating externally an initial random fraction 
of the neurons and observe the process of firing until its equilibrium. The 
final portion of neurons that are active depends in a non linear way on the 
initial fraction. The main result of this paper is a theorem which enables 
us to find the asymptotic of final proportion of the fired neurons in the 
case of random directed graphs with given node degrees as the model 
for interacting network. This gives a rigorous mathematical proof of a 
phenomena observed by physicists in neural networks [7]. 

1 Introduction 

Understanding the structure and dynamics of neural networks is a challenge for 
biologists, physicists and mathematicians. Recent experimental studies of living 
neural networks [5, 9] reveal that their global activation induced by electrical 
stimulation can be explained using the concept of bootstrap percolation on a 
directed random network. The experiment consists in activating externally an 
initial random fraction of the neurons and observe the process of firing until 
its equilibrium. The final portion of neurons that are active depends in a non 
linear way on the initial fraction. The main result shown by experiments is that 
there exists a non-zero critical value for the fraction of initially (i.e., externally) 
excited neurons beyond which the global activity jumps to a almost complete 
activation of the network, while below this critical value the firing essentially 
does not spread. The main result of this paper is a theorem which enables us 
to find the asymptotic of final proportion of the fired neurons in the case of 
random directed graphs with given node degrees as the model for interacting 
network. This gives a rigorous mathematical proof of a phenomena observed 
by physicists in neural networks [7]. Cohen et al. in [7] find this asymptotic 
via mean- field assumption and they compare it to simulations and experiment. 
The validity of the random graph approximation to metric graphs such as the 
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experimental neural networks is discussed in [18]. Bootstrap percolation model 
has been used in several related applications (see for example [2,10,16,19]). For 
a review, we refer the reader to [1]. 

A neural network is a group of interconnected neurons functioning as a 
circuit. The neural network is modeled as a directed graph [5] whose nodes 
are neurons connected by synapses. The total number of neurons is n. Let 
G = (V,E) be a directed graph on the vertex set V — [l,...,n]. We denote 
i — ► j if there is a directed link from i to j. Each node has two degrees, an in- 
dcgree, the number of links that point into the node, and an out-degree, which 
is the number pointing out. We denote by df the in-degree of node i and d~ its 
out-degree. Let A denote the adjacency matrix of graph G, with Aij = 1 if j — > i 
and otherwise. Once a neurons fired, it stays on forever. At the beginning 
of the process, a neuron has a probability a to fire as a direct response to the 
externally applied electrical stimulus and will be on at time t + 1 if at time t it 
was on, or if at least of its incoming nodes were on at time t. 

We denote by Xi(t) the state of the neuron i at time t. It is on if Xi(t) = 1 
and off if Xi(t) = 0. Then at each time step t+1, each node i applies: 

X i {t+l) = X i {t) + {l-X i {t))l^A ij X j {t)>nj , (1) 

where 1 (S) denotes the indicator of an event 5; this is 1 if S holds and 
otherwise. The dynamics is monotonic from the definition, since a firing neuron 
can never turn off and therefore Xf(t + 1) > Xi(t). Let us define & n (a, t) as 

n 
3 = 1 

We are interested to find the asymptotic value of 

$„(a) := lim $„(a,t), 

t— >oo 

when n — > oo, in the case of random directed graphs with arbitrary degree 
distribution (see for example Molloy and Reed [13,14], Janson [11], Newman, 
Strogatz and Watts [15] and Cooper and Frieze [8]) as the underlying model 
for the interacting network. Let us define P(j, k) to be the probability that a 
randomly chosen vertex has in-degree j and out-degree k. We remark that in 
general this joint distribution of j and k is not equal to the product PjPk of 
the separate distributions of in- and out-degree. Since every edge on a directed 
graph must leave some vertex and enter another, P(j, k) must satisfy Y]j k (j — 
k)P(j, k) = 0. The next section describes this model of random digraphs. 

1.1 Notation and definitions 

We are interested in constructing a random directed graph on n vertices. Let 
d-n = {{dn,i)i=i} an( ^ = {(^n.i)"=il be sequences of non-negative integers 
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such that Y%=i dni = Yl7=i d n »• The configuration model (CM) on n vertices 
with degree sequences d+ and d~ is constructed as follows: 

A vertex i is represented by the set of its incoming and outgoing edges that 
we denote by , and W~ respectively with \Wf\ = df , \W~\ = d~[ . Let 
W + = (J- and W~ = (L Wf. A configuration is a matching of W + with 
W~ and we choose the configuration at random, uniformly over all possible 
configurations. We denote the resulted graph by CM(n, d*,d~). Observe that 
the self-loops may occur, these become rare as n — > oo (see e.g. [4], [12] or [8] 
for more precise results in this direction). 

We will let n — > oo, and assume that for each n, given d^ and d~ satisfying 
the following regularity conditions: 

Condition 1. For each n e N, d^ = {{d^Ji-i} and d~ = {(d~ ti )? =1 } are 

sequences of nonnegative integers such that Yn=i i ~ SiLi^ni' an ^' f or 
some probability distribution P(j,k) independent of n, 

1. #{i : djj = j,dZ i = k}/n — > P(j,k) as n — > oo (the degree density 
condition: the density of vertices of in-degree j and out-degree k tends to 
P(j,k)); 

2. J2j k jP(h k) = J2j k kP{j, fc) =: A G (0, oo) (finite expectation property); 

3. X)"=i dni/ n — Y17=l d-n i/ n ~ ¥ ^ as n ^ oo (the average degree tends to a 
given value X). 

We consider the asymptotic case when n — ► oo and say that an event holds 
w.h.p. (with high probability) if it holds with probability tending to 1 as n — > oo. 
We shall use for convergence in probability as n — > oo. Similarly, we use o p 
and O p in a standard way. for example, if (X n ) is a sequence of random variables, 
then X n = O p (l) means that "X n is bounded in probability" and X n — o p (n) 
means that X n /n — > 0. 

1.2 Statement of result 

In this section, we state the main theorem of this work. Let Di n and D out be 
random variables with the distribution F(D in = j,D out — k) = P(j,k). We 
define the function f a (y) as follows 

f a (y) := Xy - (1 - a)E [D out l (Bin{D in , 1 - y) < 0)] . 

Let y* = y* be the largest solution to f a {y) — in [0, 1], i.e., 

y*=max{ye [0, l]|/ a (y) = 0}. 

Remark that such y* exists because / Q (0) < 0, / Q (l) = Xa > and f a is 
continuous. The main result of this paper is the following theorem. 

Theorem 1. Consider the random graph CM(n, d\ , d~) satisfying Condition 
[7J Then we have 
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1. If y* — 0, i.e., if f a ,e{y) > for all y € (0, 1], then w.h.p. 

= l-Op(l). 

2. If y* > and furthermore y* is not a local minimum point of f a (y), then 
w.h.p. 

*„(a) = 1 - (1 - a)E [1 (Bin(D m , l-y*)< O)] + o(l). 
1.3 Simulation 

Following [5,7,17], we assume a gaussian distribution for in-degree ¥(Di n = 

k) ~ exp (y ~^~2^ ^ with fc = 50 and a = 15 based on the experimental results 

[5,17]. By Theorem [1] one could see that when Di n and -D oll t are independent, 
<fr(a) := linin—Kxj $ n (a) will depend only on the distribution of in-degree Di n . 
Figure [T] shows the three dimensional representation of the final fraction of 
fired neurons, i.e. $(a), as a function of a and Q. For comparison of this 
with experiment, we refer to Cohen et al. [7] (who showed that the model and 
experiment fit very well when using the gaussian distribution of the connections). 
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Figure 1: The final fraction of fired neurons as a function of a and Q. Here 
F(D in = k) ~ exp ( ~ ( 3~ 2 I)2 ) with k = 50 and a = 15. 

1.4 Organization of the paper 

Bootstrap Percolation is studied in detail in the next section. We describe the 
dynamics of bootstrap percolation as a Markov chain in Section 12.11 The proof 
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of out main theorem, Theorem [TJ is based on the use of differential equations for 
solving discrete random processes. This was first introduced by Wormald [20]. 
We briefly discuss his method in Section 12.21 The proof of our main result is 
given in Section [3J 

2 Preliminaires 

2.1 The Markov chain 

The aim of this section is to describe the dynamics of bootstrap percolation as 
a Markov chain, which is perfectly tailored for asymptotic study. We consider 
the bootstrap percolation on CM(n,d^,d~). Let m(n) :— Y^7=i^ni denote 
the number of out-going edges in the graph. Our analysis below consists in an 
extension of the results of [3] . 

At a given time step t neurons are partitioned into fired F(t) and non-fired 
N(t). We further partition the class of non-fired nodes according to their in and 
out degree N(t) = [j J - k W' (t). At time zero, F(0) contains the initial set of 
fired neurons. We look at the system in discrete time. At time step t + f we 
have 

F(t + 1) = F(i) |J ju G N(i) such that |F(t) [~\{w G V,A VW = 1}\ > o| . 

We use a different approach based on bilateral interactions. This allows for 
a simpler Markov chain description of the system. At each step we have one 
interaction only between two neurons, yielding at least one fired. Our process 
is as follows 

• Choose an out-going edge of a fired neuron i. 

• Identify its partner j (i.e. by construction of the random graph in the 
configuration model, the partner is given by choosing an in-going edge 
randomly among all available in-going edges) 

• Delete both edges. If j is currently non-fired and it is the O-th deleted 
in-going edge from j, then j fires. 

By definition, an interaction means coupling an out-going edge with an in- 
going edge. Our system is described in terms of 

• N?' (t), < i < fl, the number of non-fired neurons with in-degree j, 
out-degree k, and i in-going edges from fired neurons at time t, 

• F^' k (t) : the number of fired neurons with in-degree j and out-degree k 
at time 

• F(t): the number of fired neurons at time i, 

• Ni n (t) : the number of in-going edges belonging to non- fired neurons at 
time i, 
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• F in (t) : the number of in-going edges belonging to fired neurons at t, 

• F out (t) : the number of out-going edges belonging to fired neurons at t. 

Obviously for j > i we have N^' k (t) = 0. Because at each step we delete 1 
in-going edge and the number of in-going edges at time is m(n), the number 
of existing in-going edges at time t will be m(n) — t and we have 

F in {t) + N in (t) = m(n) - t. 

It is easy to see that the following identities hold: 

j.k i<H 

F mt (t) = J2kP' k (t)-t, (3) 

k,j 

F(t) = Y,'' JJ ' U: - ( 4 ) 

The process will finish at the stopping time Tf which is the first time < £ N where 
F ou t(t) = 0. The final number of fired neurons will be F(Tf). By definition of 

our process (N?', F^ k ) represents a Markov chain. We write the transition 

V / i,j,k 

probabilities of the Markov chain. 

There are three possibilities for the B, the partner of an out-going edge of a 
fired neuron A. 

1. B is fired, the next state is 

Ni' k (t+l) = N^ k (t), {0<i<fl), 
F^ k {t+l) = F 3 ' k (t). 

2. B is non-fired of in-degree j, out-degree k and this is the (i + l)-th deleted 
in-coming edge and i + 1 < O. The probability of this event is — — — — 



The next state is 

Nl k {t + l) = N^ k (t)-1, 

N&it + l) = JV**(i) + l, 

Np k {t + l) = N?; k (t), (0<*'<fi, + 

F^ k {t + l) = F^ k {t). 



m(n) — t 



3. B is non-fired of in-degree j, out-degree k and this is the il-th deleted 

in-coi 
have 



in-coming edge. Then j > O and with probability ^(n^f 1 — > we 



Ni' k (t + 1) = N^ k (t), (0<i<fi-l), 
F^ k (t + 1) = F j ' k (t) + 1. 
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Let Pt denote the pairing generated by time t, i.e., P t — {e ou t, ej ra } be the set 
of edges picked before time t. We obtain the following equations for expectation 
of (jV/' ,fc (i + 1), Fi- k (t + 1)) conditioned on P t by averaging over the possible 
transitions: 



E 



E 



N^ k (t + i)-N^ k (t)\P t 



m(n) — t ' 



J ra(n) - t 



E[F j ' k (t + l)-F j ' k (t)\P t ] = 



m(n) — t 



2.2 Wormald's Theorem 

In this section we briefly present a method introduced by Wormald in [20] for 
the analysis of a discrete random process by using differential equations. In 
particular we recall a general purpose theorem for the use of this method. This 
method has been used to analyze several kinds of algorithms on random graphs 
and random regular graphs (e.g., [6], [21] and [14]). 

Recall that a function f(u\, ...,uj) satisfies a Lipschitz condition on D € W 
if a constant L > exists with the property that 

...,uj) - f(v 1 , ...,Vj)\ < L max \m - u»| 

l<t<3 

for all (ui,...,Uj) and (v\, Vj) in D. For variables Y\,...,Yt, and for D e 
the stopping time Tjj(Yi, 1^) is defined to be the minimum t such 
that (t/n;Yi(t)/n, ...,Yi,(t)/n) ^ D. This is written as Tjj when Yi,...,Y(, are 
understood from the context. 

The following theorem is a reformulation of Theorem 5.1 of [21], modified 
and extended for the case of an infinite number of variables. In it, "uniformly" 
refers to the convergence implicit in the o() terms. Hypothesis (1) ensures that 
Y t does not change too quickly throughout the process. Hypothesis (2) tells us 
what we expect for the rate of change to be, and property (3) ensures that this 
rate does not change too quickly. 

Theorem 2 (Wormald [21]). Let b = b(n) be given (b is the number of vari- 
ables). For 1 < I < b, suppose Yi(t) is a sequence of real-valued random vari- 
ables, such that < Yi(t) < Cn for some constant C, and H t be the history of 
the sequence, i.e. the sequence {Yj(k), < j < b, < k < t}. 

Suppose also that for some bounded connected open set D — D(n) C 
containing the intersection of {(t, Z\, z\f) : t > 0} with some neighborhood of 

{(0,zi,...,z h ) : P(Fi(0) = z t n,l < I < b) ^ for some n}, 

the following three conditions are verified: 
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1. (Boundcdncss). For some function [3 — f3(n) > 1 and for all t < To 

max \Yi{t + l)-Yi{t)\ < (3 
i<i<b 

2. (Trend). For some function A = Ai(n) = o(l) and for all I < b andt < Tr> 

\E[Yt(t + 1) - Y t (t)\H t ] - Mt/^Y^/n, ...,y,(t)/n)| < A x . 

<?. (Lipschitz). For eac/i Z t/ie function fi is continuous and satisfies a Lips- 
chitz condition on D with all Lipschitz constants uniformly bounded. 

Then the following holds 

(a) For (0, Zi, z^) £ D, the system of differential equations 

= fi{s,zi,...,z{), l = l,...,b, 

as 

has a unique solution in D, zi : M — > K /or Z = 1, ... ,6, which passes 
through z/(0) = I — 1, ...,&„ and which extends to points arbitrarily 
close to the boundary of D. 

(b) Let A > Ai iot£/i A = o(l). For a sufficiently large constant C, with proba- 

bility 1 - O (jf exp we We 

F/(t) = nzi(t/n) + 0(Xn) 

uniformly for < t < an and for each I. Here zi(t) is the solution in 
(a) with zi = Yi(0)/n, and a — a(n) is the supremum of those s to which 
the solution can be extended before reaching within l°° -distance CX of the 
boundary of D . 

We will also use the following corollary of the above theorem, which is namely 
Theorem 6.1 of [21]. This theorem states that, as long as condition 3 holds in 
D, the solution of the system of equations above can be extended beyond the 
boundary of D, into D. 

Corollary 3. For any set D = D{n) C R b+1 , let T b = T^JYi, ...,Y b ) be the 

minimum t such that (i, . . . , D (the stopping time). A ssume in 

addition that the first two hypotheses of Theorem fJI are verified but only within 
the restricted range t < T b of t. Then the conclusions of the theorem hold as 
before, after replacing < t < an by < t < minjern, T b }. 



Proof. For 1 < j < b , define random variables Yj by 
Yj(t + 1)-- 



Yj(t + 1) iit<T b 
Yj(t) + /j(t/n,Fi(t)/7i, Yj(t)/n) otherwise 

for all t > 0. Then the Yj satisfy the hypotheses of Theorem [3J and so the 
corollary follows as Yj(t) = Y 3 (t) for < t < T^. □ 
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3 Proof of Theorem [T] 

The proof of Theorem [1] is mainly based on Theorem [2] Indeed we will apply 
this theorem to show that the trajectory of N( ,k {t) and F^ k (t) throughout the 
algorithm is a.a.s. close to the solution of the deterministic differential equations 
suggested by these equations. 
Define 

b(n):= ]T jk(Q + l). 

l<j,k<n 

For simplicity the dependence on n is dropped from the notations. For e > 0, 
we define the domains D(e) as 



D 



(e) = {(r, K'*} i<tUfc , {f J l },.,) G R"(") +1 : - e < k < X, -e < r < X - e, 



Let Td be the stopping time for D which is the first time t when 

(t/n,{Nl k {t/n)},{F>' k {t/n)}){D. 
Let DE be the following system of differential equations: 



(nj fc )'(r) 
(nf)'(r) 

(f J ' k )'(r) 
with initial conditions 



J n o k ( T ) 

X-T ' 

(J -i + l)»-l fc i(T) - (j - i)nl' k (T) 
X-t 

X-T 



ni k = (1 - a)P(j, k), nf k (0) = for < i < fi, / 3 ' fc (0) = aP(j, fc). 
We have 

Lemma 4. 1. The system DE /ias a unique solution in D(e) which extends 
to points arbitrarily close to the boundary of D(e). 

2. For a sufficiently large constant C , with high probability we have 

Nl k (t)/n = nf k (t/n)+o(l), (5) 
F^ k (t)/n = P' k (t/n)+o(l), (6) 

uniformly for all t < na. Here a = u{n) is the supremum of those r for 
which the solution of these differential equations can be extended before 
reaching within l°° -distance Cn — of the boundary of D(e). 
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Proof. We will use Theorem O The domain D(e) is a bounded open set which 
contains all initial values of variables which may happen with positive proba- 
bility. Each variable is bounded by a constant times n. By the definition of our 
process, the Boundedness Hypothesis is satisfied with (3(n) — 1. Trend Hypoth- 
esis is satisfied by some Xi(n) — 0(1 /n). Finally the third condition (Lipschitz 
Hypothesis) of the theorem is also satisfied since A — r is bounded away from 
zero. Then we set A = O^i -1 / 4 ) > Ax. The conclusion of Theorem [2] now gives 

Nf' k {t)/n = nf fe (^)+0(n 3 / 4 ), 
F^ k (t)/n = f' k (t/n)+0{n 3 ^), 

with probability 1 — 0(n 7 / 4 exp(— n 1 / 4 )) uniformly for all t < no. Finally, for 
< i < Q; we have N?' k (0)/n = 0, and by Condition [1] and by definition: 

N^ k (0)/n A (1 - a)P(j, k), f**(0)/n A aP(j, k). 
This completes the proof. □ 

To analyze a, we need to determine which constraint is violated when the 
solution reaches the boundary of D(e). It cannot be the first constraint, because 

must give asymptotically feasible values oiN(' k and F^ k up until the bound- 
ary is approached. It remains to determine which of the last two constraints is 
violated when r = a. 

We first solve the system of differential equations (DE) and then we analyze 
the point to which the resulting equations are valid. 

Lemma 5. The solution of the system of differential equations (DE) is 
nf k (r) = P(j, a) -i/)', 

f j ' k (r) = P(j,k)[a+(l-a)F(Bin(j,l-y)>n)], 
where y = (1 — t/A). 

Proof. Let u = u{r) = —ln(X — t). Then u(0) = —ln(X), u is strictly monotone 
and so is the inverse function r = r(it). We write the system of differential 
equations (DE) with respect to u: 

(n J ' k )'(u) = -jn 3 ' k (u), 

(nf k )'(u) = (j -i + l)n{'\{u) - {j - i)nf k (u). 

Then using 

and by induction, we find 

4 k (u)=e-^ u -< ^Y l ( j ~ r \ (l-e-(»- u C0))y~ P „J.* ( U (0)). 

r-=n \ ' 
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By going back to r, we have 



(t) = y d ~ j E 4 fc (°) ( • _ J (i - f) i_r , y = (i - r M)' 



which gives 
We have 



^ fc (T) = P(j ! fc)(l-a)(- ? )2/^(l-t/)'. 



= -A 



A(/^)'(r) 
(j - n + l)njj* ! 



A// 



= -(j - n + l)P(i, fc)(l - a) ^ J Jy J - n (l - yf" 1 
= -P(j, fc)(l - a)jP(Bin(j - 1, 1 - y) = - 1). 
Then using the fact that 

— ¥(Bin(N,p) > K) = NP(Bin(N -l,p) = K), 
op 

and by initial condition we have 

f- k = P(J, k) [a + (1 - a)¥(Bin(j, 1 - y) > fi)] . 



Let us dchnc 



fout(r) := E fc ^ fc ( T )" T ' and 



/(r) = Y^hF^r:. 

Lemma 6. Assume a — <r(n) be the same as Lemma^4\ For t < no~, we 
\F ut{t)/n - fout{t/n)\ A 0, 

and 

\F(t)/n-f(t/n)\^0. 
Proof. By definition, we have 



\F out {t)/n - fout{t/n)\ 



^2k(F^ k (t)/n-f' k (t/n)) 



3,k 



< J2k\F^(t)/n-f^(t/n)\, 
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and 

\F(t)/n-f(t/n)\ < ^|^ fe (i)/n-f' fc (t/n)| 

< ^k\F^ k {t)/n- f j - k {t/n)\ . 

3,k 

It remains to show 

J2k\Fi> k {t)/n-f j ' k Wn)\ AO. 

By LemmaSl for each j and fc, we have \F^' k {t)/n — fi ,k (t/n)\ A 0. Hence, the 
same holds for any finite partial sum, which is for each K £ N, we have 

J2 j\Fi> k (t)/n-f' k (t/n)\^0. 

j,k<K 

By Condition [T] 53 . k kP (j, k) — > A £ (0, oo). Then, there exist a constant if, 
such that J2j k>K kP(j, k) < e - -By Lemma [5] 

/i.*( T ) = fc) [a + (1 - a)P(fltn(j, !-»)>")]< P(j, k). 

Let -/V(j, k) denote the number of vertices with in-degree j and out-degree k 
at time 0. Again, by Condition [IJ J^ . fc kN(j, k)/n — > kP(j,k). Hence, also 
J2 3 ,k>K kN (J' k )/ n ~* J2 3 ,k>K kP (^ k ) < e - Therefore, if n is large enough, 
E 3 ,fc>K fc7V (j' fc )/ n < e > and 

k\F^ k (t)/n-f' k (t/n)\ < k (F^ k (t)/n + f^ k (t/n)) 

j,k>K j,k>K 

< ]T k(N{j,k)/n + P(j,k))<2e. 

j,k>K 

This completes the proof. □ 

We now return to the proof of Theorem [TJ By Lemma [5] and by equation 
||7J), we have 

fout(r) = Y kP (3,k)[a + (l-a)F(Bin(j,l-y)>n)]-r 
= Xa + 53 kP(j, k)(l - a)P(Bin(j, 1 - y) > fi) - r 

= Ay - A(l - a) + (1 - a) 

E[D out l(Bin(D in ,l-y) > ft)} 

= f a (y), 

where y = (1 — r/A). 
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First assume f a (y) > for all y 6 (0,1], i.e., y* = 0. This implies Di n > £1 
almost surely. Then we have Yljk^f^ — t > Q in D(e). So the boundary 
reached is determined by f = A — e which is y — e/A. Then it is easy to see 
f out ft) = 0(e) and Lemma |5] implies \Af\ = n — 0(ne). This proves the first 
part of the theorem. 

Now consider y* > 0, and further y* is not a local minimum point of f a (y)- 
Then f a (y) < for some interval (y* — e, y*). Then the first constraint is 
violated at time f = A(l — y*). We apply Corollary [3] with D the domain D(e) 
defined above, and the domain D replaced by D'(e), which is the same as D 
except that the last constraint is omitted: 

D'(e) = {(r, {„:■*■},. {/'■'■ }^ S R fc (") +1 : - e < nf < A, -e < r < A - e, 

' • A }. 

This gives us the convergence in equations ([5]) upto the point where the 
solution leaves D'(e) or when F out (t) > is violated. Since f ou tft) begins to go 
negative after f, from equation §§§ it follows that F out (t) > must be violated 
a.a.s. and it becomes zero at some Tf ~ fn. Then by Lemma [S] and Lemma [S] 
we conclude 

F(T f ) = Y, FJ ' k ( T f) 

= n(l - (1 - a)E [1 (Bin{D ln , l-y*)< 0)]) + o p (n). 
This completes the proof. 
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